Image processing

ABSTRACT

Picture elements of a digitally coded image are each represented by a colour value. The value of an element is adjusted by firstly making a number of comparisons between the element and elements y elsewhere in the image. This comparison involves comparing a first picture element group (which comprises the picture element x under consideration and at least one further picture element in the vicinity thereof) with a second picture element group. The second picture element group comprises a base picture element y and at least one further picture element, the number of picture elements in the second group being the same as the number of picture elements in the first group and the position of the or each further element of the second group relative to the base picture element of the second group being the same as the position of the or a respective further element of the first group relative to the picture element under consideration. The comparison determines ( 114 ) whether the two groups match in the sense that they meet a criterion of similarity. When at least one comparison results in a match, a replacement colour value for the picture element under consideration is generated ( 116 ), as a function of the colour value for the base picture element of the or each second group for which a match was obtained

The present invention is concerned with image processing.

According to the present invention there is provided a method ofprocessing a digitally coded image in which picture elements' are eachrepresented by a colour value, comprising, for each of a plurality ofsaid picture elements:

(a) performing a plurality of comparisons, each comparison comprisingcomparing a first picture element group which comprises the pictureelement under consideration and at least one further picture element inthe vicinity thereof with a second picture element group which comprisesa base picture element and at least one further picture element, thenumber of picture elements in the second group being the same as thenumber of picture elements in the first group and the position of the oreach further element of the second group relative to the base pictureelement of the second group being the same as the position of the or arespective further element of the first group relative to the pictureelement under consideration, wherein each comparison determines whetherthe two groups match in the sense that they meet a criterion ofsimilarity; and

(b) when at least one comparison results in a match, computing areplacement colour value for the picture element under consideration,the replacement colour value being a function of the colour value forthe base picture element of the or each second group for which a matchwas obtained.

Preferably, the method includes identifying picture elements which meeta criterion of distinctiveness, and computing a replacement colour valueonly for picture elements not meeting the distinctiveness criterion.

Other, preferred, aspects of the invention are defined in the claims.

Some embodiments of the invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of an apparatus for performing the invention;

FIG. 2 is a flowchart of the steps to be performed by the apparatus ofFIG. 1 in accordance with one embodiment of the invention;

FIG. 3 is a similar flowchart for a second embodiment of the invention;

FIG. 4 is a similar flowchart for a third embodiment of the invention;and

FIGS. 5 to 8 illustrate the effects of this processing on some sampleimages.

FIG. 1 shows an apparatus consisting of a general purpose computerprogrammed to perform image analysis according to a first embodiment ofthe invention. It has a bus 1, to which are connected a centralprocessing unit 2, a visual display 3, a keyboard 4, a scanner 5 (orother input device, not shown) for input of images, and a memory 6.

In the memory 6 are stored an operating system 601, a program 602 forperforming the image analysis, and storage areas 603, 604 for storing animage to be processed and a processed image, respectively. Each image isstored as a two-dimensional array of values, each value representing thebrightness and/or colour of a picture element within the array.

In a first embodiment of the invention, the program 602 is arranged tooperate as shown in the flowchart of FIG. 2. The image to be processedis stored as an array [C] of pixels where the position of a pixel isexpressed in Cartesian co-ordinates e.g. (x₁, x₂) or as a vector (inbold type) e.g. x=(x₁, x₂). The colour of a pixel at x is stored as avector C(x) consisting of three components. In these examples r, g, bcomponents are used but other colour spaces could be employed. In amonochrome system C would have only one (luminance) component. Theresults of this process are to be stored in a similar array C_(OUT).

The process is iterative and starts at Step 100 which simply indicatesthat one begins with one pixel x and repeats for each other pixel (theorder which this is performed is not significant), exiting the loop atStep 102 when all have been processed. However it is not essential toprocess all pixels: some may be deliberately excluded, for reasons thatwill be discussed presently.

In Step 104, a comparison count I is set to zero, a match count M is setto 1, and a colour vector V is set to the colour at x. V has threecomponents which take values according to the colour space employed e.g.(r, g, b).

Step 106: n (typically 3) random pixels at X′_(i)=(x′_(i1)x′_(i2)) i=1,. . . n are selected in the neighbourhood of x where

|x′_(ij)−x_(j)|<r_(j) for all j=1, 2 and r_(j) defines the size of arectangular neighbourhood (or square neighbourhood with r₁=r₂=r). Atypical value for r_(j) would be 2 for a 640×416 image.

Step 108: A pixel at y =(y₁, y₂) is then randomly selected elsewhere inthe image and (Step 110) the comparison count I incremented. This pixelis selected to be >r_(j) from the image boundary to avoid edge effects.If desired, the choice of y could be limited to lie within a certainmaximum distance from x. If, at Step 112, the value of I does not exceedthe value of a threshold L (typical values are 10-100) a test for amatch between the neighbourhoods of x and y is carried out.

Step 114: Let the colour of the pixel at x be C(x)=(C₁(x), C₂(x),C₃(X))=(r_(x), g_(x), b_(x))

Then the neighbourhoods match if each of the pixels x, x′_(i)(that is,the pixel under consideration and its n neighbouring pixels) matches thecorresponding pixel at y, Y′_(i), where the positions of y′_(i) relativeto y are the same as those of x′_(i), relative to x. That is to say:x−x′ _(i) =y−y′ _(i) for all i=1, . . . , n.

where pixels at x and y are deemed to match if|C _(j)(x)−C _(j)(y)|<d _(j) for allj=1, 2, 3.

and similarly for x′_(i) and y′_(i):|C _(j)(x′ _(i))−C_(j)(y′ _(i))|<d _(j) for allj=1,2,3 and all i=1, . .. , n.

where d_(j) is a threshold that determines whether colour component j issufficiently different to constitute a pixel mismatch. In the testsdescribed below, the colour components were represented on a scale of 0to 255 and a single value of d_(j)=80 was used. In general d_(j) may bedependent upon x. For example, it may be preferred to model attention sothat less emphasis is given to darker regions by increasing d_(j) inthese areas.

If a match is found then at Step 116 the counter M is incremented andthe values of the colour components at y are added to V.V=V+C(y)

Following a match the process returns to Step 106 of selecting a freshneighbourhood around x containing n random pixels, whereas if no matchis found it returns to Step 108 to select a new y without changing thepixel neighbourhood.

If at Step 112 the value of I exceeds the threshold L, the colour of thepixel at x=(x₁, x₂) in the transformed image is given (Step 118) theaverage value of the colours of the M pixels found to have matchingneighbourhoods i.e.C _(OUT)(X)=V/M.

This process is repeated from Step 100 until all pixels in the imagehave been dealt with. The resulting transformed image possesses a muchreduced spread of colours but also contains small levels of noisearising from the random nature of the algorithm. This noise can besimply removed (Step 120) by applying a standard smoothing algorithm. Inthis embodiment a pixel is assigned the average colour of the pixels inthe surrounding 3×3 window.

The algorithm shown in FIG. 2 processes all pixels x, and all will havetheir colours altered except in the case of pixels whose neighbourhoodsare so dissimilar to the rest of the image that no matches are found. Inthat the process necessarily involves a loss of information, we preferto identify important parts of the image and exclude these. Thus theembodiment of FIG. 3 excludes regions of interest from the filteringprocess. In FIG. 3, those steps which are identical to those of FIG. 2are given the same reference numerals.

The process begins at Step 130 with the generation of a saliency mapconsisting of an array of attention scores Scores(x₁, x₂) using themethod described in our international patent application WO01/61648(also published as US 20020080133). Other methods of generating saliencymaps may also be used although their performance may not always be bestsuited to this application: See for example L Itti, C Koch and E Niebur,“A model of saliency-based visual attention for rapid scene analysis,”IEEE Trans on PAMI, vol 20, no 11, pp 1254-1259, November 1998; WOsberger and A J Maeder, “Automatic identification of perceptuallyimportant regions in an image,” Proc 14^(th) IEEE Int. Conf on PatternRecognition, pp 701-704, August 1998; and W Osberger, U.S. PatentApplication no 2002/0126891, “Visual Attention Model,” Sep. 12, 2002.The values of Scores(x₁, x₂) assigned to each pixel at x=(x₁, x₂) or asubset of pixels in an image reflect the level of attention at thatlocation.

A value is given to the variable T. typically 0.9, which sets athreshold on Scores(x₁, x₂) and determines whether the colour of thepixel at x is to be transformed or not whereThreshold=T*(max−min)+minand max=x₁,x₂Max (Scores (x₁, x₂))min=x₁,x₂Min (Scores(x₁, x₂)).However, other means of calculating the value of Threshold may be usedsome of which can be dependent upon x.

If at Step 132 the value of Scores(x₁, x₂) is greater than Threshold,the pixel in the original image at x is, at Step 134, copied unchangedinto the transformed image array C_(OUT)(x₁, x₂). This pixel representsa point of high attention in the image and will not be altered by thisprocess.

The remainder of the process is as previously described: note howeverthat, owing to the test at 132, in the smoothing algorithm, the colourvalue is replaced by the smoothed value only for those pixels whoseattention scores are less than the value of Threshold.

Another embodiment is shown in FIG. 4 in which attention scores are notcomputed beforehand. Instead, when at Step 112 the comparison count Iexceeds the threshold L, a test is performed at Step 150 to determinewhether the match count M is greater than a threshold mt. If so, then,as before, the colour of the pixel at (x₁, x₂) in the transformed imageis given, at Step 118, the average value of the colours of the Mpixelsfound to have matching neighbourhoods i.e. V/M

If, however, M is less than or equal to mt, the pixel in the originalimage at x is, at Step 152, copied unchanged into the transformed imagearray C_(OUT)(x₁, x₂). This means that pixels representing areas of highattention will be unlikely to be altered because only low values ofMwill be obtained in these image regions.

The degree of filtering that is applied to the image may be controlledby selecting the value of the thresholds d_(j). Alternatively, or inaddition, the filtering process can if desired be repeated: as shown atStep 170 in all three versions. The transformed image may be reloadedwhilst (in the case of FIG. 4) retaining the original attention scoresScores(x₁, x₂) and the whole process repeated to obtain successivetransformations and greater suppression of background information.

Note that where random selection is called for, pseudo-random selectionmay be used instead.

Once filtering is complete, the transformed image may if desired beencoded in JPEG format (or any other compression algorithm) as shown atStep 180. The reduction in information contained in regions of lowinterest enables higher compression performances to be attained than onthe original image.

The results of applying the algorithm of FIG. 4 to a football source areshown in FIG. 5, which shows, from left to right, the original image(GIF format), the image after JPEG coding, and the image after filteringfollowed by JPEG CODING. Histograms of the distribution of hue values inthe range 1-100 are also shown. It is found that the filtering reducesthe compressed image file size from 13719 bytes to 10853 bytes.

Typically two iterations of pixel colour replacement and smoothing aresufficient, but this can be extended depending upon the colour reductionrequired.

FIG. 6 illustrates how background information may be substantiallyremoved whilst preserving important features of the image such as theboat and the mountain outline. The original JPEG encoding (FIG. 6 a)occupies 13361 bytes which is reduced to 10158 bytes after processingonce and JPEG encoding the transformed version (FIG. 6 b). The outputimage is reprocessed using the same VA scores and obtains a file size of8881 bytes (FIG. 6 c). A further iteration obtains a size of 8317 bytes(FIG. 6 d).

This method may be applied with advantage to images containing artefacts(such as JPEG blocking effects). The re-assignment of colours tobackground regions tends to remove artefacts which normally possess somesimilarity to their surroundings (See FIG. 7, where the original imageis shown on the left: on the right is shown the image obtained followingprocessing with this method and subsequent re-coding using JPEG).However, artefacts which are very obtrusive and interfere with the mainsubject material will not be removed.

A further application of the method is the enhancement of figure-groundor the removal of background distractions for improved recognisability.This application is illustrated in FIG. 8 in which the background isalmost completely replaced with a constant colour and the image of thedog is much more prominent. The method could therefore be applied to theprocessing of images displayed in a digital viewfinder in a camera wherethe enhancement of subject material will assist photographers to composetheir pictures.

Essential visual information is retained in the transformed imageswhilst reducing the variability of colours in unimportant areas. Thetransformed image thereby become much easier to segment usingconventional algorithms because there are fewer colour boundaries tonegotiate and shape outlines are more distinct. This means that thismethod will enhance the performance of many conventional algorithms thatseek to partition images into separate and meaningful homogeneousregions for whatever purpose.

In the embodiments we have described, the replacement colour value usedis the average of the original value and those of all the pixels whichit was found to match (although in fact it is not essential that theoriginal value be included). Although it practice this does notnecessarily result in a reduction in the number of different colourvalues in the image, nevertheless it results in a reduction in thecolour variability and hence—as has been demonstrated—increases thescope for compression and/or reduces the perception of artefacts in theimage. Other replacement strategies may be adopted instead. For example,having obtained the average, the replacement could be chosen to be thatone of a more limited (i.e. more coarsely quantised) range of colours towhich the average is closest. Or the match results could be used toidentify groups of pixels which could then all be assigned the samecolour value.

These embodiments assume that low level segmentation algorithms shouldnot be applied to those areas in an image that merit high visualattention. Such regions are naturally anomalous and contain a highdensity of meaningful information for an observer. This means that anyattempt to segment these areas is likely to be arbitrary because thereis little or no information in the surrounding regions or elsewhere inthe image that can be usefully extrapolated. On the other hand lesssignificant parts of the image that are more extensive can justifiablybe transformed using quite primitive and low level algorithms.Paradoxically, distinctive object edges in an image attract highattention and therefore are not subjected to alteration in thisapproach. In fact the edges of objects at the pixel level in real imagesare extremely complex and diverse and would need specifically tailoredalgorithms to be sure of a correct result in each case.

The second and third embodiments of the invention offer an approach tocolour compression that makes use of a visual attention algorithm todetermine visually important areas in the image which are not to betransformed. This approach therefore possesses the significant advantagethat the process of assigning region identities does not have to addressthe difficult problem of defining edges which normally hold the highestdensity of meaningful information. Non-attentive regions are transformedaccording to parameters derived from the same VA algorithm whichindicates those regions sharing properties with many other parts of theimage. The visual attention algorithm does not rely upon thepre-selection of features and hence has application to a greater rangeof images than standard feature based methods which tend to be tailoredto work on categories of images most suited to the selected featuremeasurements. Pixels in the regions subject to transformation areassigned an average colour and increased compression obtained throughJPEG encoding or any other compression standard. Compression is appliedto the least attentive regions of the image and therefore is unlikely toaffect the perceptual quality of the overall image. The algorithm may beiteratively applied to the transformed images to obtain furthercompression at the expense of more background detail.

1. A method of processing a digitally coded image in which pictureelements are each represented by a colour value, comprising, for each ofa plurality of said picture elements: (a) performing a plurality ofcomparisons, each comparison comprising comparing a first pictureelement group which comprises the picture element under considerationand at least one further picture element in the vicinity thereof with asecond picture element group which comprises a base picture element andat least one further picture element, the number of picture elements inthe second group being the same as the number of picture elements in thefirst group and the position of the or each further element of thesecond group relative to the base picture element of the second groupbeing the same as the position of the or a respective further element ofthe first group relative to the picture element under consideration,wherein each comparison determines whether the two groups match in thesense that they meet a criterion of similarity; and (b) when at leastone comparison results in a match, computing a replacement colour valuefor the picture element under consideration, the replacement colourvalue being a function of the colour value for the base picture elementof the or each second group for which a match was obtained.
 2. A methodaccording to claim 1 including identifying picture elements which meet acriterion of distinctiveness, and computing a replacement colour valueonly for picture elements not meeting the distinctiveness criterion. 3.A method according to claim 2 wherein the step of identifying pictureelements which meet a criterion of distinctiveness is performed inadvance, and said comparisons are performed only for picture elementsnot meeting the distinctiveness criterion.
 4. A method according toclaim 2 wherein the criterion of distinctiveness for a picture elementis that the number of matches obtained for that picture element exceedsa threshold.
 5. A method according to claim 1 wherein the selection ofthe or each further picture element of the first group is selected in arandom or pseudo-random manner.
 6. A method according to claim 1 whereinthe selection of the or each further picture element of the first groupis selected in a random or pseudo-random manner from picture elementslying within a predetermined distance of the element underconsideration.
 7. A method according to claim 1 wherein the furtherelements are selected afresh following a match.
 8. A method according toclaim 1 wherein the selection of the base picture element of the secondgroup is selected in a random or pseudo-random manner.
 9. A methodaccording to claim 1 wherein the selection of the base picture elementof the second group is selected in a random or pseudo-random manner frompicture elements lying within a predetermined distance of the elementunder consideration.
 10. A method according to claim 1 wherein the basepicture element for the second group is selected afresh for eachcomparison.
 11. A method according to claim 1 wherein the image is amonochrome image and the colour value is a single, luminance component.12. A method according to claim 1 wherein the image is a colour imageand the colour value has three components.
 13. A method according toclaim 11 wherein the match criterion is that no component of any pictureelement of the first group differs from the corresponding component ofthe spatially corresponding element of the second group by more than athreshold amount.
 14. A method according to claim 1 wherein thereplacement colour value for a pixel is a function also of its existingcolour value.
 15. A method according to claim 14 wherein the replacementcolour value is the average of the colour value for the picture elementunder consideration and the colour value for the base picture element ofthe or each second group for which a match was obtained.
 16. A methodaccording to claim 14 wherein the replacement colour value is that oneof a predetermined set of colour values which is closest to the averageof the colour value for the picture element under consideration and thecolour value for the base picture element of the or each second groupfor a match was obtained.
 17. A method according to claim 1 includingthe step of processing the processed image again.
 18. A method accordingto claim 1 including the step of applying spatial filtering to theprocessed image.
 19. A method according to claim 1 including the step ofencoding the processed image using a compression algorithm.
 20. An imageprocessing apparatus comprising means (603) for storing an image andmeans (2, 602) arranged in operation to perform the steps of claim 1.21. A computer program comprising a data carrier having stored thereon aset of instructions for performing the method of claim 1.